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ANALYSIS  OF  OFFICER  PERFORMANCE  ON  AN  EXPERIMENTAL  TASK- 
COMMUNICATIONS  EXHIBIT 


BACKGROUND 

S*j 

The  Communications  Exhibit  Task  Is  one  of  15  situational  performance 
exercises  developed  and  administered  as  a  segment  of  a  large-scale  longi¬ 
tudinal  research  project  initiated  in  response  to  recommendations  by  the 
Army  Scientific  Advisory  Panel  (ASAP)  and  the  Deputy  Chief  of  Staff  for 
Personnel  (DCSPER).  The  former  indicated  a  need  for  additional  research 
on  the  performance  and  selection  of  combat  officers  and  suggested  that 
dimensions  of  such  performance  might  be  defined  by  means  of  performance 
exercises  within  a  combat  simulation.  DCSPER,  in  view  of  the  increasing 
complexity  of  military  technology,  was  interested  in  determining  the  feasi¬ 
bility  of  differential  prediction  of  performance  for  broad  areas  of  possi¬ 
ble  officer  specialization.  The  research  design  incorporates  both  sets 
of  requirements.  The  differential  prediction  aspect  of  the  research  is 
concerned  with  three  broad  areas --combat,  administrative,  and  technical.^ 
Experimental  predictor  tests  relevant  to  these  areas  were  administered  to^\ 
6900  officers  on  entrance  to  active  duty  in  I958  and  1959*  and  a  revised 
battery  to  4000  on  entrance  to  active  duty  from  1961  to  1964.  Fifteen 
to  thirty  months  later,  a  subsample  of  900  of  the  latter  group,  six  at  a 
time,  were  assigned  to  the  Officer  Evaluation  Center,  Fort  McClellan, 

Alabama,  established  for  the  purpose  of  providing  criterion  data  on  offi¬ 
cer  performance.  There,  in  a  simulated  Military  Assistance  Advisory 
Group  (MAAG)  setting,  over  a  period  of  three  days,  a  scenario  unfolded 
which  eventuated  in  invasion  and  guerrilla  warfare.  Each  group  of  six 
officers  received  a  series  of  assignments,  first  administrative  and  tech¬ 
nical,  and  then  combat.  Performance  was  recorded  and  rated  out  of  sight 
of  the  examinee  by  cadre  who  played  the  parts  of  MAAG,  host  nation,  and 
aggressor  personnel.  Work  products  were  retained  for  later  scoring.  The 
performance  records  and  work  products,  after  analysis  to  define  underlying 
dimensions,  serve  as  criteria  for  the  predictor  tests. 

The  Communications  Exhibit  Task  was  one  of  the  five  in  the  technical 
area  and  was  performed  on  the  first  day.  It  was  designed  to  measure 
trouble-shooting  ability  and  ability  to  use  effectively  technical  facili¬ 
ties  and  manuals  and  the  services  of  an  enlisted  assistant.  The  examinee's 
job  was  to  diagnose  and,  if  possible,  to  get  into  working  order  an  assem¬ 
blage  of  telephone  equipment  which  was  to  be  used  shortly  to  demonstrate 
to  host  nation  officers  the  operation  of  a  field  network.  The  examinee 
had  two  hours  to  complete  the  task.  He  was  given  a  circuit  diagram  and 
requisite  manuals  and  an  inexperienced  enlisted  man  was  assigned  as  his 
assistant.  Some  spare  equipment  was  on  hand.  When  the  examinee  discovered 
but  could  not  correct  a  defect,  he  was  to  record  the  symptom  and  the  loca¬ 
tion  of  the  trouble  and  recommend  corrective  action. 

The  equipment  and  interconnections  contained  fifteen  defects  or  "bugs, 
including  missing,  defective,  or  wrongly  inserted  parts,  missing  or  inac¬ 
curate  line  tags,  and  short,  open,  or  misconnected  circuits.  Repairs  made 


by  the  examinee  and  the  examinee's  notes  provided  the  basis  for  the 
entries  on  the  main  scoring  record,  the  Trouble  Scoring  Sheet.  At  the 
end  of  the  two  hours,  the  officer  in  charge,  ostensibly  to  prepare  him¬ 
self  to  assist  the  signal  officer  in  demonstrating  the  equipment,  ques¬ 
tioned  the  examinee  concerning  operation  of  the  equipment.  The  subject 
officer's  responses  were  scored  unobserved  by  the  enlisted  man  on  a 
Question  Scoring  Sheet.  Various  aspects  of  the  examinee's  performance 
and  various  personal  characteristics  were  rated  at  the  end  of  the  task 
by  the  enlisted  man  and  by  the  officer  in  charge,  the  latter  basing  his 
judgment  in  part  on  observations  through  a  one-way  screen. 


OBJECTIVES 

The  main  objectives  of  the  present  analysis  were  to  obtain  dimen¬ 
sions  (factors)  summarizing  the  performance  and  behavior  represented  in 
the  scoring  records  on  the  Communications  Exhibit  Task  and  to  provide 
scores  for  major  parts  of  the  scoring  record,  the  dimensions,  and  the 
task  as  a  whole.  These  scores  will  be  correlated  with  scores  fiom  the 
other  14  tasks  to  indicate  the  extent  to  which  each  score  is  specific  to 
the  task,  common  to  the  tasks  of  the  technical  area,  and  general  across 
all  three  areas.  From  these  scores  and  those  of  the  other  tasks,  crite¬ 
rion  scores  will  then  be  derived  to  validate  the  experimental  predictor 
tests. 


METHOD 


SAMPLE 

The  sample  consisted  of  846  officers  for  whom  Cummunications  Exhibit 
scoring  records  were  available.  For  ratings  cf  motivation  and  attitude, 
which  were  introduced  after  the  test  had  been  administered  to  several 
groups,  only  726  cases  were  available,  and  for  ratings  introduced  still 
later,  on  factors  that  the  rater  would  consider  if  he  were  making  an  over 
all  evaluation,  694  cases. 


VARIABLES 

Principal  variables  are  listed  in  Table  1.  These  variables,  except 
total  score  (to  be  described  later),  were  derived  from  four  scoring  forms 

1.  Trouble  Scoring  Sheet.  For  each  of  the  15  "bugs"  in  the  appara¬ 
tus,  labeling,  or  circuits,  a  checkmark  indicated  which,  if  any,  of  the 
acceptable  corrective  actions  was  taken  by  the  examinee;  if  he  did  not 
correct  the  defect,  checkmarks  indicated  whether  he  had  recorded  the 
symptoms,  indicated  the  location  of  the  trouble,  and  recommended  appro¬ 
priate  corrective  action.  Scores  of  three  types  were  obtained:  for  each 


corrective -act Ion- taken  alternative,  a  1,  0  score;  for  each  "bug",  a  score 
from  0  to  4,  1  for  recording  the  symptom  only,  2  for  recording  location, 

3  for  reeonmending  appropriate  corrective  action,  and  4  for  taking  correc¬ 
tive  action;  and  a  total  score. 

2.  Questions  Scoring  Sheet.  Checkmarks  indicated  which  of  various 
creditable  answer*  were  given  to  the  questions  of  the  officer  in  charge. 
Three  scores  were  obtained:  1)  1,  0  score  for  each  answer,  2)  score  for 
each  question  (answer  credits  were  summed  when  the  question  required  more 
than  one  creditable  answer),  and  3)  total  score.  In  computing  questions 
and  total  scores,  double  weight  was  given  to  certain  answers  judged  espe¬ 
cially  important  or  superior  to  alternative  answers  which  merited  credit. 

3.  Problem  Approach  Checklist.  Three  variables  were  rated  by  both 
the  enlisted  man  and  officer  in  charge:  problem  approach  (the  extent  to 
which  the  examinee  was  systematic  in  undertaking  his  task),  effective  use 
of  the  enlisted  man,  and  extent  of  use  of  the  manuals  (appropriate,  too 
great,  or  too  little).  An  option  in  rating  on  use  of  the  manuals  pro¬ 
vided  for  identification  of  examinees  who  were  judged  sufficiently  famil¬ 
iar  with  the  equipment  that  they  did  not  need  to  use  the  manuals.  Because 
of  close  agreement,  only  ratings  by  the  officer  in  charge  were  scored 
except  when  the  form  was  incomplete.  (Each  rating  option  was  scored  zero- 
one,  and  each  of  the  three  main  variables  zero-one  or  zero-one-two. ) 

4.  Descriptive  Report  II.  This  report  form,  completed  by  both  the 
officer  in  charge  and  the  enlisted  man,  required  the  rating  of  motivation 
and  attitude,  each  on  a  five-point  scale  and  scored  1  to  3,  and  indication 
of  which  of  ten  listed  factors  seemed  particularly  pertinent  to  evaluation 
of  the  subject  officer's  overall  performance.  Seven  of  the  factors  are 
listed  In  Tables  1  and  2;  the  other  three  are  Effective  Expression, 
Endurance  and  Stamina,  and  Other  (to  be  specified  by  the  rater).  Also 
indicated  was  whether  the  officer  was  considered  strong  or  weak  on  a 
selected  factor.  Each  of  the  listed  factors  was  scored  0  if  designated 

as  a  weak  point,  2  if  designated  as  a  strong  point,  and  1  if  neither. 

Three  sets  of  Descriptive  Report  scores  were  obtained,  one  for  the  offi¬ 
cer  in  charge,  one  for  the  enlisted  assistant,  and  for  the  two  summed. 


ANALYSIS 

Scores  at  the  sub-item  level  (specific  corrective  actions  for  each 
"bug, "specific  answers  to  questions,  and  specific  rating  steps  on  the 
Problem  Approach  Checklist)  were  intercorrelated  (product -moment  r's) 
together  with  Trouble  Scoring  total.  Question  Scoring  total,  and  Descrip¬ 
tive  Report  factor  total.  The  objectives  were  to  evaluate  sub-item 
validity  by  means  of  correlation  with  totals,  to  identify  among  sub-items 
having  low  correlation  with  totals  any  clusters  suggestive  of  atypical 
but  valid  content,  and  to  evaluate  the  desirability  of  and  appropriate 
ratios  for  weighting  of  sub-items. 


After. changes  indicated  by  the  analysis,  scores  at  the  item  level 
(specific  "bugs",  specific  questions,  and  specific  rating  scales,  includ¬ 
ing  only  those  of  the  10  rated  factors  that  were  indicated  as  important 
on  at  least  of  the  forms)  were  intercorrelated,  together  with  Trouble 
Scoring  total.  Question  Scoring  total,  and  Descriptive  Report  factor 
total.  Objectives  were  to  evaluate  item  scores  in  the  same  manner  as 
scores  at  the  sub-item  level  and  to  provide  a  basis  for  factor  analysis. 
The  Judgment  as  to  whether  the  examinee  was  sufficiently  familiar  with  the 
equipment  that  he  did  not  need  to  use  the  manuals,  from  Descriptive  Report 
II,  scored  0,  1,  was  included. 

The  factor  analysis  was  by  the  principal -component  method.  Results 
were  rotated  by  the  varimax  procedure.  Scores  for  factors  were  estab¬ 
lished  by  selection  of  high  loading  variables  with  relatively  low  ioadings 
on  other  factors. 

Reliability  of  major  components  was  estimated  from  element  and  com¬ 
posite  variances  by  means  of  Cronbach's  alpha.  Reliability  of  motivation 
and  attitude  ratings  was  evaluated  by  correlation  between  ratings  provided 
by  the  officer  in  charge  and  those  provided  by  the  enlisted  assistant. 

For  use  in  cross-test  and  other  analyses,  a  total-score  formula  was 
established,  consistent  with  the  following  considerations: 

1.  Measures  of  performance  would  be  preferred  to  measures  of 
personal  attributes. 

2.  Measures  of  accomplishment  would  be  preferred  to  scores 
representing  procedures  or  means. 

3*  Objective  scores,  if  sufficiently  comprehensive,  would  be 
preferred  to  subjective. 

4.  In  the  absence  of  an  Independent,  more  nearly  ultimate  criterion, 
differential  weighting  would  not  be  undertaken  for  fairly  homogeneous  items 
nor  to  replace  natural  weights  of  internally  homogeneous  item  groups  whose 
respective  size  appeared  reasonably  proportional  in  importance  to  the  per- 
formance  represented. 

3.  The  composite  should  cover  all  major  parts  of  the  task. 

6.  High  reliability  (if  not  achieved  through  undue  reduction  in 
comprehensiveness)  was  desirable. 


RESULTS 

ITEM  SCORES  AMD  INTERCORRELATIONS 

On  the  basis  of  sub-item  intercorrelations,  a  few  changes  were  made 
la  the  Initially  established  item  scores--ellmlnation  of  credit  for 
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certain  corrective  actions  having  lower  correlation  with  composites  and 
appearing  less  desirable  on  rational  grounds,  and  minor  alteration  of 
weights  for  Trouble  Shooting  Approach  and  Use  of  Manuals  alternatives. 

Three  rated  factors  were  eliminated  from  further  analysis  largely 
becauae  of  infrequent  indication  of  importance  and  low  variance,  but 
also  because  there  was  only  minimal  requirement  in  this  task  for  the  first 
two  of  the  traits  represented.  These  three  rated  factors  are  Effective 
Expression,  Endurance  and  Stamina,  and  Other  Factors  (to  be  specified  by 
the  rater).  No  other  variables  had  variance  so  small  or  p-values  so  ex- 
treme  that  elimination  seemed  appropriate.  Intercorrelations  of  remaining 
Items  and  of  totals  are  shown  in  Table  1. 


RELIABILITY 

The  Cronbach  alpha  reliability  coefficient  for  the  Trouble  Scoring 
Sheet  total  was  .84  and  for  the  Question  Scoring  Sheet  total  .63. 

Correlation  coefficients  obtained  between  ratings  given  by  the 
officer  in  charge  and  by  the  enlisted  assistant  on  motivation  and  atti¬ 
tude  were  .73  and  <70,  respectively.  Coefficients  were  not  obtained  for 
Problem  Approach  Checklist  variables  nor  for  the  Descriptive  Report 
factors  considered.  ; 


FACTOR  ANALYSIS 

The  varimax  rotation  at  the  9“factor  level  was  accepted.  Fifty-four 
percent  of  the  total  variance  was  accounted  for  (1.2$  more  than  by  the 
8-factor  solution),  and  three  to  eleven  percent  by  each  of  the  rotated 
factors.  The  nine  factors  are  identified  below.  For  each  factor,  the 
higher-loading  variables  which  contribute  to  its  definition  and  serve 
also  as  a  factor  score  are  designated  by  item  number,  if  any,  and  listed 
in  order  of  the  magnitude  of  the  loading,  which  is  given  in  parentheses. 

1.  Equipment  "bugs.” 

2.  Battery  missing  from  a  phone  (.68). 

4.  A  circuit  selector  switch  in  wrong  position  (.65). 

1.  A  missing  transmitter  unit  (.63). 

13.  A  defective  transmitter  unit  (.34). 

2.  Motivation  and  attitude.  (Preference  was  given  to  the  composite 
ratings  of  officer  and  EM,  though  loadings  were  slightly  lower  than  for 
officer  alone.) 
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Attitude  composite  (.82). 

Motivation  composite  (.78). 

3.  Questions  on  equipment  functions. 

2.  Functions  of  a  terminal  box  (.61). 

3.  Reasons  for  tagging  lines  (.55). 

6.  Advantages  of  using  5-Pair  rubber  cable  (.44). 

4.  Circuit  "bugs." 

8.  A  misconnection  at  a  terminal  (.67). 

14.  An  open  trunk  (.65). 

9.  A  shorted  line  (.63). 

15.  Line  attached  to  wrong  binding  posts  C.55)*  ' 

5-  Familiarity  with  equipment. 

Factor  considered--familiarity  with  equipment  (.73). 

Knew  equipment  so  did  not  need  manuals  (.70). 

6.  Tagging  "bugs." 

7.  A  missing  tag  (.62). 

6.  A  mistagged  line  (.60). 

7-  Utilization  of  enlisted  man. 

Utilization  of  personnel  (.56). 

Factor  considered — ef fective  command  and  control  (.48). 

8.  Questions  on  circuits. 

1.  Whether  any  phone  in  the  net  can  call  any  other  (.58). 

4.  Alternate  route  for  a  specified  call  (.47). 

7.  Circuit  modification  required  for  a  specified  call  (.46). 
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9.  Bearing. 


Faccor  considered--bearing  and  assurance  (.51). 

Factor  considered--keeping  cool  (.43). 

Factor  considered --general  impression  (.36). 

The  Appendix  presents,  for  items  and  their  sums,  all  loadings  of  .20 
or  higher  on  the  9  factors.  Communal i ties  ranged  from  .14  (Bug  #3>  a 
defective  signal  lamp)  to  .99  (Trouble  Scoring  Sheet  total). 


TOTAL  SCORE 

The  total  established  for  use  as  a  comprehensive  score  In  examining 
relationships  of  the  test  with  outside  variables  consisted  of  the  standard 
score  on  the  Question  Scoring  Sheet  total  plus  three  times  the  standard  * 
score  on  the  Trouble  Scoring  Sheet  total.  This  sum  was  converted  to  a 
score  with  a  mean  of  500  and  standard  deviation  of  100.  The  estimated 
reliability  of  the  composite  was  .86  vs  .63  and  .84  for  the  >two  component 
scores,  respectively. 
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APPENDIX 


FACTOR  LOADINGS 
(Of  .20  and  greater) 


Variables 

Factors 

1 

2 

*  5 

6 

.1  . 

8 

9 

Trouble  Scoring  Sheet.  Bugs: 


1 

.63 

2 

.68 

3 

.27- 

4 

.65 

5 

•34 

9 

6 

.60 

7 

.62 

8 

.67 

9 

.63 

•  33 

10 

.41 

•  30 

11 

•33  .20 

•  59 

12 

•  35 

.48 

13 

•  52 

'  >30 

14 

.65 

.20 

15 

.26 

•  55 

Total 

.60  .21 

.61 

CM 

• 

.28 

Question  Scoring  Sheet.  Questions: 

1 

2  .61 

3 


4 


23  -55 

.50  .22 


APPENDIX  (Continued) 


FACTOR  LOADINGS 
(Of  .20  and  greater) 


Variables 

Factors 

1 

2 

3 

4 

5 

6 

7 

8 

9 

5 

•25 

.29 

.20 

6 

.44 

.23 

•27 

7 

.20 

.23 

.22 

.46 

Total 

.22 

.61 

.26 

•55 

Problem  Approach  Checklist 

Troubleshooting  Approach 

•  39 

.25 

.22 

.22 

•  36 

•  30 

Utilization  of  Personnel 

.27 

■56 

Use  of  Available  Manuals 

•  32 

•  25 

•25 

.42 

•  31 

Knew  equipment  (did 
not  need  manuals) 

■70 

Motivation 

e 

Officer  in  charge 

.22 

.81 

•25 

Officer  and  EM 

•25 

•78 

.28 

.21 

Attitude 

Officer  in  charge 

.86 

Officer  and  EM 

.82 

.20 

.22 

Factors  Considered 

Bearing  and  assurance 

.22 

.31 

.30 

•51 

Keeping  cool 

.22 

•  43 

Familiarity  with  equipment 

•  23 

.29 

•75 

Following  instructions 

.24 

.27 

•  37 

•  59 

Mission  accomplishment 

.46 

•  53 

.34 

Effective  command  and  Control 

.48 

General  impression 

.36 

Total 

•  36 

.28 

.20 

•  *5 

.42 

.39 

•39 
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